I've read through all those numbers, and I can't point out much that's clearly wrong - but it's definitely giving off a bad code smell. One thing I can clearly point out is this:
Q: What is the value of "Queue Timeout Settings" (in the ColdFusion Administrator)?
A: 560 seconds; this value was defined by the dev team, this is because we have processes that take a long time to be executed, according to what the dev team indicates this delay is normal. If for example, we lower this value to 60 seconds, timeout errors begin to be reported by system users.
560 seconds is a REALLY HIGH value for any global timeout setting. If I ran into this as a load tester, and I didn't know anything else about the code, I'd try to identify the specific reasons why it was originally set this high. If a page or module takes this long to execute, why is that? Is it happening across the board, or are some pages and modules taking this long while most aren't? What do those pages and modules share in common? Are they interacting with specific database tables, or performing operations that are known to be long-running like PDF generation, or calculating reports that are more complex than the rest of the application? I dunno, but I'd look for that. If you could set a long timeout just for these pages, and a short default timeout for everything else, that would probably improve performance and reduce memory use significantly.
Here's another potential bad practice.
Q:Does your application use Application.cfc? If so what are the values for:
A:
sessionmanagement="Yes"
clientmanagement="yes"
sessiontimeout="#CreateTimeSpan(0,6,0,0)#"
setclientcookies="true"
That is a REALLY LONG session timeout - SIX HOURS! A session is normally kept for thirty minutes or less, the lower the better. Unfortunately, HTTP doesn't have a built-in session tracker so you have to guess how long users will wait around before clicking a button or a link, etc. Sessions are stored in memory, and this is going to use a lot more memory than required. If a user comes to your site, does something in the first hour, then closes the browser window, your server will still keep that session around for another five hours! So, what you might think of as not memory-intensive may turn out to unnecessarily hold onto memory it doesn't need to. Try setting this to thirty minutes or less in a test environment, and see what happens. If your developers say "we need it to be this long because of blah blah blah" that's a good sign you need new developers.
The rest of my objections are just code smells like I mentioned before, but this application seems to need a LOT more memory than, well, nearly any other CF app I've seen. I'd be really interested to see how much memory you actually need. You shouldn't need a 64GB max heap size unless there's some heavy-duty stuff going on. I really doubt you'd have 250 concurrent users if you had reasonably short sessions.
I'd probably disagree with or at least question some of the other advice you've received. For example, if you increased the min heap size to 64GB as well, that's going to take a while to restart - my wild guess is twenty minutes or more. That's why you get two different values to set. If your max heap isn't that large, or if you have a cluster of web and/or application servers, it will often make sense to set the min heap the same so there isn't so much reallocation going on after startup, but if your max heap IS that large and you have a single application server, you probably don't want all that allocation going on before the machine can start working. But that's my opinion and just a minor quibble. I would definitely start reducing the actual memory allocation needed during the app's runtime. Performance analysis load testing will be your friend here, unless you like developing on production. (Seriously, don't develop on production.)
Finally, and this is sort of unrelated, you want to stop using searchImplicitScopes=true as quickly as possible. This is a critical security issue. Yes, you will have to go through your entire codebase to fix this. It's a giant pain. But there's some help out there! Pete Freitag's Fixinator tool can find and fix these unscoped variables, and Adobe has released a patch that will log these variables. You can find out all about that by scrolling to the bottom of this page:
https://www.petefreitag.com/blog/cf-searchimplicitscopes/
Dave Watts, Eidolon LLC
... View more